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ABSTRACT 

The Department of Defense (DOD) has undertaken major 
research and development programs to improve the quality of their 
documentation, yet there continue to be major deficiencies in the 
quality of the manuals produced. In an attempt to identify factors in 
the design and development process that may be contributing to such 
deficiencies, this project studied five publication houses, four of 
which produced DOD technical manuals. Writers, managers, and 
illustrators were interviewed at each site to obtain information on 
the work flow in developing a manual; the use of specifications; the 
validation and verification process; and the skills and duties of 
writers, illustrators, and government representatives. This report 
presents an introductory review of the literature; a description of 
the research methodology; and discussions of the findings on 
specification use and deficiencies, cost of inadequate/inappropriate 
specifications, relative effort in the workflow, the government's 
technical manual representative, writers and their qualifications and 
training, writers' job duties, source materials, degree of 
automation, comprehensiveness, illustrations, quality assurance, 
validation, and verification. A final discussion of the study results 
and their implications, a list of 29 references, and a flow chart of 
a typical technical information development process conclude the 
report. (THC) 
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Abstract 

The Department of Defense has undertaken major research and development programs 
to improve the quality of their documentation, yet there roninues to be major deficiencies in 
the quality of the manuals produced. Five publication houses, four of them producers of 
DOD technical manuals, were studied in ail attempt to identify factors in the design and 
development process which may be contributing to manual deficiencies. Writers, managers, 
and illustrators were interviewed at each site. The work flow in developing a manjal is 
described. Data on the use of specifications, the validation and verification pro.cess. and the 
skills and duties of writiers. illustrators, and government representative are presented. 
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Technical documentation has risen to a new position of recognition and even 
prominence in our society. The usability of the technical documentation is widely recognized 
as a critical factor in how well we can use the flood of new technology and it is 
increasingly recognized as an important variable in marketing and sales. 

What has led to this new found prominence of the technical manual? We wouid 
propose that it is due to the increasing numbers of users !hat are novices to the equipment. 
Technology used to be in the hands of a few "experts" who had worked with the particular 
equipment or similar equipment for years. Furthermore, there was a pool of coexperts to 
be called upon whenever a problem arose. Thus most of the system knowledge was in the 
head of the user. The documentation was simply a formal, engineering oriented description 
of the system which could serve as a reference. 

Today however, the typical user of the system is a "novice" - a person with 
inadequate knowledge to effectively operate and maintain a system without the support of 
documentation. What happened to create this new "typical" user? First, and most obviously 
complex technology became affordable and applicable in our daily activities. Hence, large 
numbers of individuals totally naive to the particular technology became users of that 
technology. Personal computers are a clea' example of this development of a naive user 
group but there are many other instances. For example, office personnel must maintain very 
complex reproduction equipment and bank tellers must maintain the automatic teller 
machines. 

There is another kind of novice however. This individual is an expert in the broad 
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domain in which the technology is applied, but is a novice with regards to the particular 
equipment. While the absolute novice described above is what one typically thinks of when 
considering audience, we would predict that this expert who is a novice regarding the 
particular equipment will be (if not already) the prevalent user for which we will be designing 
documentation. This novice-expert (or is it an expert-novice) has resulted from the dramatic 
advances in technological development. In the past an individual could become an expert 
and maintain his or her expert status because there was little variation in the family of 
equipment and the technology evolved gradually. Today however the technology is changing 
so rapidly that it is virtually impossible to maintain expertise (in the old sense) in some 
areas. For example, it is assumed that the knowledge a new engineer brings to the job will 
only be adequate for 10 years - at which point the technology will have evolved to a point 
where retraining will oe required. But even before one's knowledge becomes obsolete, the 
variations within a family of equipment (.e.g., minicomputers, programmable controllers, 
reproduction equipment, etc.) makes it difficult to transfer that expertise from one equipment 
to another. 

Whereas the expert of old may only have required system documentation, the novice 
and the expert-novice require quidance in carrying out particular operation and maintenance 
tasks. That is, they require task-oriented or functional manuals in order to perform their job. 
The information must be organized and presented at a level of detail and in a format that 
facilitates performance of the job. However, the traditional, and still prevalent, design for 
manuals is simply a complete system description from the engineer's perspective one 
section describes the theory of operation and successing sections describe the physical 
components of the equipment. The contrast between the user's needs and these traditional 
manuals is between an organization based on system functions rather than hardware 
components and a presentation oriented toward the job tasks rather than toward engineering 
theory (Duffy. 1985b). 

6 
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While the need for user oriented manuals aopears to be a relatively recent 
phenomenon, it has been the focus of Department of Defense research and development 
efforts for over 35 years (Smillie, 1985), This is because the Department of Defense has 
long been confronted with the problem of a large body of novice users (at least by civilian 
standards) of technology. A typical enlistment is for only four years During that time, the 
individual must be trained in the military structure as well as in a technical skill and then 
must make a positive contribution through the application of the technical skill in the 
operation or maintenance of some of the most sophisticated equipment in the worlo. 

Freeman, Hubler, and Monroe (1980) interviewed a random sample of 32 men who 
were working in aircraft maintenance (aviation mechanics, structural and hydraulic) and 
provided the following profile. The typical mechanic is 24 years old, has a high school 
diploma, never enjoyed reading, had a ninth grade reading ability upon graduation from high 
school but through non-use reads well-organized test at almost the se/enth grade level, does 
not use the traditional technical manuals for much of his work preferring to use 
proceduralized, task specific, maintenance repan cards, and gives no indication that he 
understands how or why the system works. Furthermore the average score on the general 
classification test (a measure of general ability where the mean is 50 and the standard 
deviation is 10) was approximately 49%. 

Given this profile and the short period of employment it is obviously essential that the 
job guidence (i e , (he technical documentation) be easy to use and understand. Sntivcr and 
Hart (1975), used he research data on document design strategies to estimate the cost 
savings to the U.S. Army possible through effective document design. They estimated that 
$830 million would be saved due to reduced maintenance time and a reduced number of 
false removals of functioning parts. A further $900 million in reduced training time would be 
saved if the manuals were fully proceduralized. 

• / 
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Recognizing the need for more effective manuals, the Department of Defense has 
developed rules, guidelines, task analysis procedures, validation and verification procedures, 
and a host of specific user or task oriented organizations and formats in a continuing effort 
to improve the usability of the manuals (Hatterick and Price, 1981; Duffy, 1985a: Kern. 1985: 
Smillie. 1985: Little and Smith. 1983). Hatterick and Price (1981) describe over one hundred 
user oriented design techniques thai have been developed since the 1950's. Experimental 
evaluation of these strategies has indicated very significant time savings and improved 
accuracy when these job-oriented manuals aie used (see, e.g., Potter and Thomas, 1976: 
Elliott and Joyce, 1968) 

Given all of the above it is rather astonishing to find that the technical manuals 
prepared for the Department of Defense seem to be little improved. There are still horror 
stones of the inadequacies of the manuals and of the rejection of the manuals by 
maintenance prersonnel (Duffy. 1985a. b; Smillie, 1985; General Accounting Office, 1979: 
Kern, 1985: Sulit and Fuller. 1976). The General Accounting Office (1979) describes one 
radar manual where the technician had to refer to 165 pages in eight documents and look 
in 41 different places in those documents to repair one particular malfunction. Clearly, the 
manual is not task oriented. 

Above and beyond the design issues, the manuals are notorious for the complexity of 
the writing. They are jargon ridden above and beyond the shorthand jargon the technician 
would normally use, the sentences have a bureaucratic complexity, and even the non-jargon 
vocabulary is overly complex and abstract. (Kniffin, et. al., Kincaid. Fishourne, Rogers, and 
Chisssom. 1975, Duffy. 1985a). The problem is such that the Department of Defense has 
begun to impose readability standards and controlled vocabulary lists and is moving toward 
automated authoring techniques (Frase. McDonald, and Keenan. 1985. Braby and Kincaid. 
1981; Kincaid, Aagard. O'Hara. and Cottrell. 1981 
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These comprehendibility problems persist in the technical manuals in spite of the fact 
that the solution involves following the basic tenants of clear writing that have been known 
for centuries. The development of a rhetoric of explanatory text has a long history (Conners. 
1985). There are innumerable style guides and virtually all trumpet the need to consider the 
audience and condemn the overly complex syntax and vocabulary. Furthermore, the writers 
of the military technical manuals are classified as professional .Titers and have been trained 
or have extensive experience in wnting. 

In sum. we have a case where there is extensive research and guidance on the 
design of technical manuals and manuals are being developed by professional writers, 
yet there remain significant problems in the quality of the manuals. Our objective in this 
3tudy was to begin an analysis of the technical manual production process to identify those 
factors contributing to the difficulties in producing a manual. We did this through inte. yiews 
with writers, illustrators, managers, and other personnel at five publication houses, While 
there has been extensive research on the individual writing process, there is little data on 
the publication system and how the writer fits into that system. Our basic assumption is 
that the poor manuals arise from some set of inadequacies in the publication process and 
that if there is to be an improvement in the quality of the manuals that improvement will 
only arise from a modification in the production system. 

We are not denying the critical importance of the tn.u and financial constraints placed 
on publication groups. Document development is notoriously underfundec. especially if the 
ha r 1ware development group controls the budget (Jordan. Kleinman and Shimberg, 1981. 
Duffy, 1985b) The start of a manual production effort is too frequently delayed until the 
product is almost developed and then a rush job is required to meet the delivery deadline 
Those time and money constraints, like all other constraints, have specific effects on „.c 
publication process. We need to understand the publication process and the factors within 
that process that contribute to inadequacies in the manuals. We must understand the 



process and the inadequacies of the process if we are to improve the system - regardless 
of whether or not the time and money constraints persist. We might also note that time and 
.money just may not be the major factors hindering the development of quality manuals (see. 
e.g.. Duffy, Curran, and Sass, 1983: Wright. 1985; Flower and Hayes, 1981). 

We already have data describing the workflow in the development of a technical 
manual. Deardorff. Hagemen. Hehs. and Norton (1979) completed a quality assurance study 
which included the determination of the typrcal technical information development process. 
These findings, presented in Figure 1. indicated 30 distinct work activities shown as blocks 
in the figure. Within the 30 work activities Ihere are four key events: preparation of the 
draft manual (blocks 1-10). validation (block 11), verification (block 21). and printing and 
distribution (blocks 29 and 30). In addition to the internal workflow, there is also the 
interaction with the government specification and the government representative monitoring 
and approving the work. 

Insert Figure 1 about here 



We used the Deardorff, et. al.. flow chart to guioe our own interview process. We 
wanted to validate their finomgs and extend the findings to consider what management, 
writer, illustrator, and government activities during each event may be contributing to the 
problems of manual usability. We interviewed personnel at four publication houses that focus 
on Defense publications We attempted to get a representative sampling of the puohcation 
houses and therefore looked for both small specialty houses as well as large corporations 
where the focus is on producing manuals for equipment they are developing. We also 
looked at houses developing new manuals and houses involved in updates and revisions. 
Finally, we included one publication house that did not develop government manuals but 
rather they produced manuals for products developed mhouse or by sister companies We 
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suspected government requirements may be a major tactor in the manual development 
problems and therefore thought the inclusion of this nongovernmentcompany might suggest 
alternative publication strategies. 

Method 

The basic approach involved conducting open-ended structured interviews with 

management and line personnel at tha production houses. 
Subjects. 

Personnel from five production, houses were interviewed. Four C the production 
houses prepared military TMs and one prepared civilian TMs. The houses preparing military 
TMs differed in size: one was small, two were of intermediate size, and one was large. 
The civilian production house was large. 

The production houses were in three locations in the Middle Atlantic states and on the 
West Coast. The work of the single small production house typically involved writing TMs 
from vendor's literature, updating technical drawings, and producing illustrated parts 
breakdowns. The primary work of one of the medium-sized companies was in rewritirg 
military TMs that had been found to be deficient. The other medium-sizea company and 
the large companies prepared TMs for equipment/systems which they or other companies 
had under development. 

Interviews were held with a variety of personnel dt each site. The positions held by 

interviewees ranged from top level management to new writers and illustrators. At each 

production house persons interviewed included at least one first-level manager, a lead writer 

or a writer, and an illustrator, or a person knowledgeable about the illustration area. The 

specific individuals to be interviewed were selected by the company. At each company, the 

number of personnel from which responses were gathered ranged from three to 20 
Interview Form 

A structured interview form was developed and used to guide the interview sessions 
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The items were selected based on experience of project personnel, discussions with the 

subject matter experts, and the findings of othar relevant projects. There were 26 questions 

covering six areas: relative effort within the workflow, management, writers, illustrators. 

validation, and quality assurance (QA). 
Interview Sessions 

The interview was open-ended to allow coverage of topics not anticipated in the 
questionnaire. Items from the questionnaire were posed and the respondents answered at 
whatever length and depth they wished, including discussion of topics not on the 
questionnaire. The questionnaire was used as a checklist by the interviewer so that all 
topics listed on it would be cohered and would form a common basis across all interviews. 
The order of. the discussion was not believed to be important and it varied from person to 
person. 

There were three types of interview situations. In the first, discussions were held with 
a group of interviewees; the second type began with a group and led to individual 
interviews; and the third type involved a series of individual interviews. The different 
approaches were required because of a /ailability of personnel and production hojse 
preferences. 

Results 

The interview data from each subject was summarized into key points and issues 
relevant to either describing the production process or identifying key factors that affect the 
quality of the technical manual. The subjects comments could be sorted into eight topic 
areas: (1) specifications, (2) workflow. (3) government's TM representative, (4) writers (5) 
illustrators. (6) qual : ty assurance (7) validation, and (8) verification The findings for each 
topic area are presented below. 
Specifications 

It is the contractor who develops the TM, but Government actions during pre-contract 
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award period have important implications to TM comprehensibility. It is during this period 
that the Government chooses the specifications which help drive the design and cost of the 
final product. The discussion below covers interviewee comments on the spec!' cations. 

Specifications used. When asked what specifications appear most frequently in the 
RFPS and contracts, all interviewees answered that MIL-M-38784 was the major and most 
fiequently seen specification. For Naval 3ea Systems Command, MIL-M-15071G (SHIPS), 
Miiitaiy Specification: Manuals, Equipment, and Systems, is frequently a companion 
document to MIL-M-38784 These specifications are for the development of standard, topic 
oriented, technical documents. The subjects also indicated several specifications for 
nontraditional manuals which they have had to write to: MIL-M-24100B, Functionally 
Oriented Maintenance Manual (FOMM); MIL-M-81701. MIL-M-81919. MIL-Vs-81927. or MIL- 
M-81928 for Work Packages; and MIL-M-63038, Army's "New Look" Skill Performance Aids. 
These specification* are for functional and user oriented documents and contain significantly 
more information on what content is to be included and how that content is to be organized 
and formatted. There has been research evaluating the effectiveness of each of these 
specifications and in each case designing to the specification led to a significant 
improvement in performance (Hatterick and Price, 1981; Duffy, 1985b). 

The publication house producing nonmilitary documents is also guided by specifications. 
There are two primary specifications: one for the development of "standard" manuals and 
one for the development of highly graphic, procedural manuals. 

Specification D eficiencies. The civilian and military publication houses stand in sharp 
contrast in terms c* their evaluation of the specifications. Subjects at the civilian house had 
very few complaints about the specifications. There are at least two reasons for this 
acceptance of the specification. First, there is little variation in the kind of manuals that are 
developed. Thus the specification can be specific to the particular type of equipment and 
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job - it does not have to be applied to a wide variety of equipment types. Second, new 
writers attend a "school" in which they are trained to write to the specification. 

The publishers of military manuals, in contrast, indicated that the specifications, as 
they are used, are seriously deficient and contribute to both the cost and the difficulty in 
using the TMs. According to interviewees, the deficiencies include: (1) an indiscriminate 
application of the specifications to all types of technology (equipment). (2) no interpretation 
of the specifications. (3) an indiscriminate application of the specifications over all 
maintenance levels, and (4) inadequate specifications for new technologies. 

Regarding the indiscriminate applications of specifications, production house personnel 
complained that there was no attempt on the part of the Government's TM representative to 
determine the portions of a specification which were applicable to the item under question. 
The Government's TM representative always invoked the entire specific i instead of 
tailoring the specification to the item under consideration. For example, even the standard 
specification calls for a functional orientation in the maintenance section where the operation 
of the equipment is described. Most electronic systems lend themselves to this type of 
presentation, e.g., the laser rangefinder on a tank can be described in functional terms. 
However, for many mechanical systems, this approach is not appropriate. 

For the second deficiency, no specification interpretation, the respondents indicated 
that while the specifications tell production houses what is needed, they do not specify or 
direct how to do it. Production house personnel believe that the Government should know 
the technology and the TM user well enough to interpret the specifications f or them or to 
supply them with a "Book Plan" on how to implement the specifications. The Book Plan 
would identify the specifications that are applicable, define how the specifications are to be 
used for this particular TM, and demonstrate how the TM is to be organized and 
consolidated. In essence, the writers, illustrators, and managers all have difficulty 
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interpreting how a general guideline (the specification) applies in particular circumstances. 
There is one specification (Joyce, Chenzoff, Mulligan, and Mallory, 1973) where there are 
companion reports providing just such strategy guidance for both managers and writers. 
There is no evidence that such guidance improved the use of the specification. 

The third RFP deficiency was that specifications are applied indiscriminately across all 
levels of maintenance. In the military, there are three levels of maintenance: organizational, 
intermediate, and depot These levels reflect a progression in the difficulty of the 
maintenance task and the expertise of the personnel typically assigned to the tasks. The 
compiaint is that a specification which calls for a particular level of detail is typically invoked 
for all maintenance tasks. Yei the level of detail is only appropriate for a particular level of 
maintenance. For example, in an electronics system; organizational personnel focus on the 
function of components or boxes and the interrelationship between them. They do not need 
detailed descriptions of what is occurring inside a box. In' contrast, the detailed information 
is of central concern to the depot personnel, while detailed descriptions of the 
interrelationships of the boxes is of little use. Specifications calling for different levels of 
description discussions are needed for different maintenance levels. The respondents believe 
that the existing mismatch between the user's need and the requirements of the 
specification significantly decreases the usability and the comprehendibility of the TMs. 

The fourth deficiency proposed by the writers was that the current specifications are 
not adequate for new technologies. For example, most specifications call for functional 
presentation. Yet many of the new built-in-test equipments (BITE) co not test the system 
equipment by funct* >n. Rather, 8ITE may test by card or by communication paths. New 
specifications need to be developed on how best to present thi? technical information. 

Cost of inadequate/inappropriate specifications. There is a relationship between 
cost and non-applicable portions of specifications. The production house cost estimates are 
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based on the contents of the specification. When an entire specification is applied to a 
system, the production house must base its costs on the assumption that the Government 
will insist the TM include all items required by the specifications, irrespective of whether that 
documentation is really needed. A production house proposal could point out non-applicable 
portions of the specifications, but thisi action runs the risk of being considered non- 
responsive. 

Relative Effort in the Workflow, 

Subjects were shown the production workflow chart in Figure 1 and asked to describe 
how their production deviated from that flow. They were also asked to discuss how the 
work load is distributed across the flow chart. We allowed the subjects to impose their own 
structure on the workflow when they were discussing the proportion of the workload. Thus 
different groups segmented the flow chart differently. However, we did require them to 
impose actual percentage estimates when they spoke in generalities, e.g., if a subject said, 
"most of the work occurs during stages 1 to 6" we required him or her to estimate the 
percentage. 

All production house personnel reported that their in-hour^ workflow was very similar to 
that described in the Deardorff et al. report (1979), as shown in Figure 1. Regarding the 
relative efforts expended during the workflow, the large production house that generates 
military TMs estimated that about 60 percent of total man-hours are expended in blocks 1-5 
(preparing the draft text), and another 15-20 percent is expended in block 6 (preparing the 
draft art work). The medium-sized production houses estimated that about 75 percent of 
their man-hours are expended in blocks 1-10 for electronics systems, but only 55-60 percent 
for mechanical systems. This is because mechanical systems require larger validation efforts 
than electronic systems. The remaining 25-40 percent of effort is spread out over blocks 
11-30 (validation to customer distribution). 

Personnel at the small production house estimated that 50 percent or the effort goes 
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toward writing and 50 percent toward all other activities (such as art preparation, diagrams, 
artwork, and typing) to produce a camera-ready copy when applicable. Ten percent of the 
total effort is dedicated to revisions according to customer verification and comments. 

The civilian production house produces two major types of TMs. one primarily text and 

the other primarily diagrammatic work instructions. Regarding the effort expended in 

collecting source data, it was estimated that 15 percent is required for the text TM an 20 

percent for the diagrammatic work instructions. It was further estimated that tor generating 

the initial draft an additional 40 percent of the effort is required for the text TM and 30 

percent for the diagrammatic TM. A field test consumes 10 percent of the effort for the 

text TM and about 20 percent or the effort for the diagrammatic TM. While the percentages 

vary from project to project, 't was estimated that, in general, 65 percent for text TM and 

80 percent for diagrammatic TM of the work effort has been expended by the end of block 

11. The remaining effort percent is distributed across blocks 12 through 30. 
Government's TM Representative. 

The production of the TMs is done under contract even at the publication house 
producing civilian manuals. At that house the contract is with subdivisions within the 
corporation. In spite of this contractual relationship, there was virtually no mention of a 
contract monitor or a progress review by the subdivision letting the contract. The publication 
house seems to assume responsibility for quality and progress reviews, maintaining the time 
and cost schedule, and establishing contact with tha appropriate subject matter experts. 

In contrast, when the contract is with the government, an individual or group within the 
government \s assigned responsibility for developing the request for proposals (including the 
specification) for the TM. reviewing and evaluating proposals, and monitoring the contractors 
work in developing the TM. 

When asked how the TM representative's performance could be improved, production 
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house personnel responded that he should: 1.) review the developing product more often; 2.) 
perform more front-end analysis on the specification; 3.) be more technically qualified and 
less format-oriented; and 4.) know and understand the user better. 

The subjects indicated that after contract award, the government representative meets 
with them in an initial planning session and then at periodic in-process reviews. These 
reviews are infrequent, typically linked to a percentage of the work completed, e.g. when 25, 
50, 75, and 90 percent of the draft work has been completed. 

Some subjects estimated contact as occurring only once every three or four months 
They indicated that this infrequent contact often had a negative affect on the quality and 
the efficiency of the production effort. More frequent monitoring would reduce needless 
effort by the production house on a product that was not going to be acceptable. 

The request for more front-end analyses relates to their earlier complaint that 
specifications are indiscriminately and inaccurately applied over different types of 
technologies, activities, and users. Production house personnel believe they could provide a 
better manual if some of the requirements' uncertainty were eliminated from the RFP 
through front-end analysis. 

The subjects indicated a belief that in most cases the government representative does 

not understand the equipment for which the manual is being written and is unfamiliar with 

the skills of the personnel and the circumstances under which they work. They suggested 

that the inadequate training and knowledge of the government representative is at the base 

of many of the problems. A better-trained TM representative would contribute to better TMs. 
Writers 

Generally, writers are organized into writing groups headed by a lead writer The lead 
writer is responsible for coordinating the group's activities, seeing that schedules are met. 
serving as a quality assurance point, and training the less experienced writers. The most 
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frequent comments were made about the writer and the writing process covering the 
following seven topics. 

Qualifications/Training. The production houses specializing in military work do not 
formally certify writers. Writers are hired on the basis of their technological expertise. They 
are predominantly ex-military technicians or college graduates in engineering., particularly 
electronic or electrical engineering. With the technological advances that are being made, 
the trend has been to hire fewer ex-military personnel and more engineers, and also to hire 
them immediately after graduation. Currently, the writer mix between ex-military and 
engineers was estimated to be between 50/50 and 40/60. The reason given for the trend 
toward engineers is that ex-military personnel do not possess the skills and knowledge 
applicable to advanced technology. The exception to the above trend occurs in those 
production houses that rewrite existing manuals. Their writers tend to be mostly ex-military 
personnel who have usually had experience with the item under rewrite. 

All training is on-the-job for the military production house writers. There is no formal 
program for in-house or out-of-house training. Normally, the lead writer is responsible for 
on-the-job training. 

Writer qualifications and in-house training courses are used at the civilian production 
house. These writers must have a Bachelor of Science degree in Engineering Technology 
(BSET). In past years wien there was a shortage of qualified personnel, individuals with 
two-year technical school degrees were hired and then sent to school to obtain the BSET 
degree. Today, writers must possess the BSET degree to be considered for employment. 
In-house training for the civilian production house includes courses on task analyses, 
formatting, standards, and software to aid the editing process. 

Job Duties. Major job duties for the writer include the gathering of source data, 
learning about the system, generating the text, updating the text as the design changes. 
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coordination with team members and ensuring that the text is validated. The writers for 
military TMs report that their most important, most difficult, and most time-consuming task is 
to learn about the system. The difficulty of the learning task and the subsequent writing 
task is directly related to the technology involved and the extent and accuracy of the source 
materials. This statement does not apply to those writers (probed; ex-military) who are 
already familiar with the item or system. 

Source Material. The writer's source materials include drawings, test specifications, 
engineering reports, and. when available. Logistic Support Analysis (LSA) data. The LSA is 
data identifying the maintenance and other requirements for keeping the system operational. 
While the LSA data is supposed to be generated during the product design, it is frequently 
the case that the data is not developed until the manual is under, development. Using 
these various data sources, the writer attempts to infer how the system works. The writer 
may also rely to some extent on assistance from peers and design engrneers. In some 
cases, he may have access to the equipment itself. At times, however, writers must learn 
how the system works without ever seeing it. When the actual equipment is not available, 
more errors occur in the TM because of faulty inferences, because the source material does 
not reflect final design changes, and because of lack of validation opportunities. 

When asked about LSA product deficiencies, subjects stated that the data does not 
attend to the traditional human factors issues relevant to the design of technical 
documentation. These issues include criticality of the task, head/book tradeoff, frequency of 
the task, environment in which operation/maintenance would be performed, and the skill and 
knowledge requirements of assigned personnel. 

The civilian production house writers report that obtaining source material is their most 
difficult task. The second most difficult task is trying to understand the system. The 
source materials to aid them in understanding the system include drawings, test 
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specifications, an engineer assigned to the writing group, design engineers, and, in some 
cases, prior documentation. The civilian production house writers also report that some of 
their work includes correcting and updating existing manuals. The hardware for these 
manuals is being used in the field and personnel experienced with the system act as an 
additional data source. 

Writing. Typically, one lead writer supervises several other, less experienced writers. 
Writing and assignments are done by functions of the equipment not by maintenance or 
other tasks. One or more functions are selected from the system and assigned to a group. 
It is the responsibility of the writer or group to understand and then write about the 
assigned function. Responsibility is for the entire function, whether the writing is m one 
manual or is in chapters across several manuals. 

When asked what model they write to, the writers indicated that they use the 
production house's most current TM that met the specifications listed in the contract. 

The writer leads in print media and the illustrator follows. The more the manual relies 
on text-graphic format, the more the writer and illustrator work together. When lustrations 
are called for, it is the writer who first roughs out the picture for the illustrator. The writer 
is responsible for seeing that both text and illustrations are correct. 

Degree of A utomation. All interviewees indicated that they make some use of word 
processors in generating text. All indicated that they were trying to automate the writing 
process as much as possible and as quickly as possible. One of the medium-sized 
companies and the large military production house had their word processors programmed 
for computing reading grade levels. The civilian production house was the most fully 
automated. Writing was done at the word processors with computer-aided editing, the text 
was electronically stored, and the final draft was put on a paper medium via a computer- 
controlled phototypesetting process. A video display terminal for the preparation of graphic 
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presentations was also used. 
Comprehendibilitv 

The military production houses indicated that they only attended to issues of ease of 
comprehension if specific requirements were called out in the specification. Thus, if a 
specification required that the text meet a particular readability standard and that passive 
voice not be used, then the writers would meet those specification requirements. In essence, 
the specification and a previous text meeting that specification are what guides the writers 
activities. Some of the more experienced writers, especially ex-military writers, were 
concerned that the focus on meeting the specification along with the increasing tendency to 
hire engineers fresh out of school could lead to a loss of perspective cn the users needs. 

Writers and managers at the civilian production house quite uniformly indicated they 
consciously attended to the user's needs. Writers indicated they review their drafts with the 
question in mind, "Can the person use it?" This is quite different from the "Does it meet 
the specification?" attitude expressed by the writers of military TMs As an example H this 
sensitivity to the needs of their TM users, writers at the civilian products house refer^d to 
their preparation of feature documents, TM products whose format was developed after an 
intensive survey of user preferences. 
Illustrators. 

None of the military production houses have a formal certification process for 
illustrators. They are hired based on their expertise and credentials. The writer, not the 
illustrator provides the first rough draft of a drawing. Working from the rough draft, the 
illustrator prepares the graphic in accordance with the appropriate specification (for example, 
in drawing the electrical wiring equipment symbols for ships' plans, drawings must conform 
to MiL-STD-15-2) Illustrators usually obtain their training at junior colleges, community 
colleges, or private institutions. 
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The illustrators for the civilian production house must be former draftsmen or have an 

art background. All graphics for the TM are prepared by the illustrator using an interactive 

graphics terminal, a CRT with a lightpen and a keyboard. The illustrations are stored in 

memory and put into paper form by a computer-controlled phototypesetting process. In- 

house training on this system is required. 
Quality Assurance ( QA). 

Quality assurance is the quality review which occurs during the development of the 
TM. Most specifications require a quality assurance plan indicating at what stages reviews 
will occur and what factors will be considered in the- reviews. 

All of the publication houses indicated that the quality assurance seldom includes a 

requirement for an independent group of engineers or writers to evaluate a writer's work. 

Some houses use an independent editorial QA group which checks such things as paragraph 

subordination, split infinitives, and syntax. They do not check the technical content or 

usability. QA for technical correctness of content is achieved through validation. 
Validation. 

Validation is the review of the final draft of the manual for technical accuracy and 
completeness. Production house employees said that they are fairly free to conduct 
validation in any way they wish. Typically, the contractor is required only to certify that the 
TM has been validated and is accurate. 

Validation is yenerally the responsibility of the writer. It can range from table-top 
validation in which someone reads through the manual to ah informal process in which each 
writer validates his drafts using peer engineers/technicians (familiar with the system cjt not 
with the item under test) to perform the tasks while the writer observes and supervises. 

Interviewees reported that actual TM users are rarely used during validation because of 
the increased time and cost. The only occasions in which actual TM users participate in 
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validation is when the RFP calls for it (and it rarely does) or when validation and verification 
are combined. 

All interviewees believe that the contractor decides how to conduct validation, including 

the type of subjects to use if any. Interviewees also reported that formal tests of usability, 

accessibility, and comprehensibility are not part of the validation process. 
Verification. 

Verification is the assessment of whethei the intended audience of the manual can in 
fact use it to carry out operation and maintenance tasks. While validation is the responsibility 
of the contractor, verification is meant to be the final assessment by the government of the 
adequacy of the product (manual) they are buying. 

Personnel at the companies producing government manuals indicated that there are 
few true verifications by the Government. That is, there are few times when the contractor 
does not provide any assistance and only Government personnel are used: it was estimated 
that this occurs no more than 5 percent of the time. The concern of the contractor is for 
the accuracy of the material. When the contractor is present and the TM user raises a 
question, the contractor is anxious to answer the question and to show the user how that 
information can be gotten from the TM. While the intentions of the contractor are good, 
providing that type of aid masks comprehensibility and usability deficiencies in the TM. 

Validation and verification are intended to be independent events. In many cases. 

however, the validation and verification events are held concurrently to save time and money. 

In practice this means that the contractor is more active in validation/verification than would 

be the case if verification were held separately, and the Government relics more on the 

contractor under these circumstances than it otherwise would. 

Discussion 

The reserch findings are based on interviews with publication house personnel and as 
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such they are subject to a variety of reporting biases. We hope future research will attempt 
to validate the findings through observation and assessment of the actual publication 
process. We think such efforts are critical to validating our findings and to developing an 
objective understanding and analysis of the process. Given the restrictions, the data 
nonetheless suggest three factors contribute to the difficulties i. producing effective technical 
manuals: the adversary role with the government, the qualifications of writers, and the quality 
assurance and validation process. Only the last factor is strongly influenced by time and 
money. Thus the typical explanation of time and money as the culprits in the development 
of effective manuals is. at best, only partially warranted. 

Both the government monitor and the specification system appear to be at the base of 
the governments contribution to ineffective manuals. The specification is the primary vehicle 
for controlling the design and the production of the manual. Both the production houses and 
the government use the specification as the primary tool for evaluation. One of the primary 
complaints about the specifications is that they are inappropriately applied to equipment 
types and maintenance requirements. That is, the claim is that different equipments (e.g.. 
electronic vs hydraulic; replaceble modules vs no such modules) require different guidelines 
for the organization and presentation of information. Furthermore, personnel performing 
basic maintenance tasks require a different level of detail than personnel totally overhauling 
the equipment. 

The military has recognized this need to target particular needs and has developed a 
variety of alternative specifications. Indeed, the Defense Logistics Analysis Agency (1977) 
identified 480 different specifications, standards, and instructions for the creation of manuals. 
In a further attempt to target specifications to particular needs, the Navy has just created a 
data base management system for specifications so that all specifications are readily 
accessible and appropriate parts of existing specs can be combined to create the 
appropriate match of the spec to the equipment and task. 
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Our findings would suggest that the Navy approach will only lead to the further 
proliferation of specifications and will have little effect on the quality of the manuals. There 
are already over 100 specifications for the design of user oriented manuals (Hattenck and 
Price, 1981). Some of those specifications are extremely detailed (Joyce, et.al.. 1973). The 
problem does not appear to rest in finding the appropriate specification but rather in the 
fact that the specifications tend to be the only expert design knowledge available in the 
production of technical manuals. The specification approach is simply inadequate to serve as 
the sole or even the primary mechanism for controlling the quality and usability of manuals. 

Specifications attempt to identify in an objective manner the surface characteristics 
(writing style, level of detail, organization, format, etc.) of an effective manual. The 
presumption is that if the writers and designers simply replicate those characteristics the new 
manual will also be effective. However, the specifications only address the manual as a 
product: it does not describe how to produce that product, i.e., the process of writing and 
designing, nor does it provide a rationale for the particular design strategies to guide the 
writer in adapting the particular requirements to specific writing and design situations. 

There is a significant body of research and a long history of experience indicating that 
this product oriented approach of providing rules and guidelines (as in a specification, style 
guide, or format guide) is an inadequate means of controlling the quality of the product 
(Duffy, 1981; Flower and Hayes, 1981; Hayes, Hower, Shnver, Sfatman. and Carey, 1985). 
1985) Rather than focusing on products this research suggests that the focus must be on 
the expertise of the writer and on the process of writing. This is because writing is an ill 
defined problem; it is a task that is never replicated. The context of writing - the 
constraints in the information available, the particular communication goal, and the text 
already written and to be written - is constantly changing. Because of this changing context, 
rules and procedures cannot be rigidly applied. Rather the writing must be guided by less 
rigid design heuristics and by an understanding and analysis of how the information will be 
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used. 

The specification must contain a clear description of the goal of the overall design 
and a description of how each of the details of the specification contributes to that goal. 
Some of the details might simply be exemplars of a variety of strategies for achieving a 
particular goal, e.g,, syntactic requirements to achieve readability. Other requirements may 
require rigid application to achieve consistency across a family of manuals, e.g.. particular 
strategies for showing a more detailed view of a portion of a Ifne drawing. The writers and 
designers can then adapt tne requirements as necessary to achieve the particular goal. 

Of course meeting the goals of the specification requires expertise from both the 
government monitor and the writers and designers (Smillie. 1985). The interview data suggest 
that little expertise is available. The manager is typically not a trained writer nor a 
technician. Therefore his or her review focuses on compliance with the objective statements 
in the specification. Furthermcjre, (he reviews tend to occur later in the production effort, 
when there are drafts to evaluate, rather than in the early planning stages when the 
strategies are developed. 

The writers tend to come from an engineering background. It was only at the civilian 
publication house that there was any indication that writing and design skill was an issue in 
the hiring of technical writers. Indeed, technical knowledge was considered the most 
important selection criterion, more important then even knowledge of the job context. A 
manager at one military publication house indicated that he recognized that the "writers" 
were very poor writers. However, he described their job as getting the technical data down 
- the editor is the one who makes sense of it. This again reflects the product oriented view 
of writing as simply matching grammar, format, and style requirements. 

The focus of the writers we interviewed was on interpreting the specification. They 
indicated that writing would not be a problem if only the specification clearly told them what 
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to do. However, as Smillie (1C85) has argued, it is unreasonable to expect a person not 
trained or experienced in writing and designing user oriented texts, to be able to make 
intelligent decisions regarding the level of detail, the amount of explanation, etc. The writer 
cannot comply with a specification if he or she does not understand that the intent of the 
specification is to meet the needs of the user. And only one of the four military publication 
houses indicated any attempt to interview intended users of the documentation or assess the 
job context in which the manual would be used. 

In contrast to the military publication houses the civilian house focused less on 
technical expertise and more on writing skill. Their approach would seem to reflect a 
growing trend in the hiring of writers outside of the military context. Computer firms, 
especially are more frequently hiring writing experts and sending them to school, if 
necessary, to learn about the particular equipment. 

Finally, the data indicated little attention to quality assurance either during the 
development of the manual or in the final evaluation. Accuracy and completeness were the 
issue rather than ability to use the document. However, accuracy and completeness were 
typically assessed through an expert review. There are a growing number of studies to 
indicate that such a table top review, even by writing and design experts, does not seem to 
be adequate to insure that tho necessary information is presented in a usable manner. 
Duffy, Curran, and Sass (1983) found that expert redesign of technical manuals did not 
improve the ability to locate or use the information in the manual - even when there was 
no cost or time constraint on the revision effort. Similarly, Swaney, Janik. Bond, and Hayes 
(1981) found that expert review and revision by a writing expert was an inadequate means 
for assessing the usability of the information in a document. When they asked expert writers 
to revise consumer contracts only one of four contracts was improved. They found 
improvements in the usability of the documents only when the review and revision was based 
on actual testing of the ability of users to use the document. This user testing, or protocol 
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aided revision, is being increasingly recognized as a critical step in the production of usable 
documents (Soderston, 1985; Shriver. 1984). 

In summary, the focus in the design of technical manuals must shift from the product 

to the process of developing the manuals and to the goals of the design process. 

Specifications should include goal statements and interpret design requirements in 'terms of 

those goals. Furthermore writers and managers must have design expertise to interpret the 

specifications. This may be achieved through selection or through a training and certification 

process. Further research will be required, however, to determine the specific training 

requirements or selection criteria. This writing environment clearly presents new and 

interesting issues in the teaching of writing (or the selection of writers). It is a highly 

cooperative writing environment, an issue just beginning to receive attention in composition 

research. In addition it requires a writer to accomodate demands for a high level of 

expertise in three domains: the technical area, the job context, and m writing and design 

skills. Again, -we are only beginning to consider the role of subject matter knowledge in 

writing and revising. We would anticipate that a proper training program would lead to 

greater attention to the user and job context in both planning and testing a manual. 

However, it might well be that more cost efficient user testing procedures will be required. 

References 

Braby, R. and Kincaid, J. P. Computer aided authoring and editing (TN I-8I). Orlando, 
FL: Training Analysis and Evaluation Group, February, I98I. 

Deardorff, D.D., Hageman, K.C., Hehs, W. t and Norton, J.M. Development of a quality 
assurance methodology for the technical information generating subsystem of NTIPS 
(NAVAIRSYSCOM 78-C- i075-0003). Washington, DC: Naval Air Systems Command, June 
I979. 

Defense Logistics Agency. Technical documentation staff study report. Washington, DC. 
Defense Logistics Agency, April I977. 



ERLC 



2S 



27 

Duffy, T\M. Organizing and utilizing document design options. Information Design 
Journal 1981. 2, 256-266. 

Duffy, T.M. Readability formulas: what's the use? In Duffy, T.M. and Waller, FL eds. 
Designing usable texts. New York: Academic Press, 1985a. 

Duffy, T.M. Preparing technical manuals: specifications and guidelines. In Jonassen. 
D. The technology of the text. Vol. 2. Englewood Cliffs, NJ: Educational Technology 
Publications. 1985b. 

Duffy, T M.. Curran. T.E., and Sass, D. Document design for technical job tasks: an 
evaluation. Human Factors , 1983, 25(2) , 143-160. 

Elliott, T.K. and Joyce. R.P. An experimental comparison of procedural and 
conventional electronics troubleshooting (AFHRL TR-68-1). Wright-Patterson AFB, OH: Air 
Force Human Resources Laboratory, November I968. 

Flower, L. and Hayes, J.R. Plans that guide the composing process. In Frederiksen. 
C.H. and Dominic, J.H., eds. Writing: the nature, development, and teaching of written 
communication . Volume 2. Hillsdale, NJ: Lawrence Erlbaum, I98I. 

Frase, L.T., Macdonald, N.H., and Keenan, S.A. Intuitions, algorithms, and a science of 
text design. In Duffy, T M. and Waller, R., eds. Des i gning usable texts. New York: 
Academic Press, I985. 

Freeman, D., Hubler, S., and Monroe, B. USN P-3 pilot J PA program user description 
(TR 1404). Burbank, CA: Lockheed-California Company, June I980. 

General Accounting Office. Improved Management of Maintenance Manuals Needed in 
DOD (LCD 79-105). Washington, DC: U.S. General Accounting Office. July I979. 

Hattenck, G.R. and Price, H.E. Technical order managers reference data (AFHRL TR 
80-51). Wright-Patterson AFB, OH: Air Force Human Resources Laboratory, May 1981b. 

30 



28 

Hayes, J.R., Flower, L, Schriver, K., Stratman, J., and Carey, L, Cognitive processes 
in revision (CDC TR 12). Pittsburgh, PA: Communications Design Center. Carnegie-Mellon 
University, April 1985. 

Jordan, S,, Kleinman, J., and Shimberg, H.L , eds. Handbook of technical writing 
practices . Vol. I. New York: John Wiley and Sons, 1971. 

Joyce, R.P., Chenzoff, A.P., Mulligan,W., and Mallory, W. Fully proceduralized job 
performance aids. Vol.1. II, III (AFHRL TR-73-43). Wright-Patterson Air Force Base, OH: Air 
Force Human Resources Laboratory, December, 1973. 

Kern, R.P. Modeling users and their use of technical manuals. In Duffy. T.M. and 
Waller. R., eds. Designing usable texts. New York: Academic Press, 1985. 

Kincaid, J, P., Aagard, J,, O'Hara, J,, and Cottrell, L, Computer readability editing 
system (TN 4-81). Orlando, FL: Navy Training Analysis and Evaluation Ground, April 1981. 

Kincaid. J. P.. Fishburne, R.P., Rogers, and Chissorrv, B,S. Derivation of new readability 
formulas (Automated Readability Index, Fog Count, and Flesch Reading Ease Formula) for. 
Navy enlisted personnel(Research Branch Rep. 8-75). Millington, TN: Naval Air Station, 
February 1975. 

Kniffin, J.D., Stevenson, C.R., Klare, G.R., Entin, E., Slaughter, S., and Hooke, 
L Operational conseguences o_f a literacy gap (AFHRL TR- 79-22). Brooks Air Force Base, 
TX: Air Force Human Resources Laboratory. November, 1979. 

Little, R.Q., and Smith, M.S. Improving FOMM Troubleshooting. Technical 
Communication. First Quarter, 1983, 20-24. 

Potter, N.R. and Thomas, D.L. Evaluation of three types of technical data for 
troubleshooting (AFHRL TR-76-74). vVright-Patterson AFB, OH: Air Force Human Resources 
Laboratory, September 1976. 



31 



29 

Schriver, K.A. Revising computer documentation for comprehension: ten exercises in 
protocol-aided revision (CDC TR 14). Pittsburgh, PA: Communications Design Center. 
Carnegie-Mellon University, November 1984. 

Shriver. E. and Hart, F. Study and proposal for the improvement of military technical 
information transfer methods. Aberdeen Proving Grounds, MD: U.S. Army Human Engineering 
Laboratory. December 1975. 

Smillie, R.J. Design strategies for job performance aids. In Duffy, T.M. and Waller. R., 
eds. Designing usable texts. New York: Academic Press. 1985. 

So^jrston. C. The usability edit: a new level. Technical Communication. First Quarter. 
1985, 16-18. 

Sulit, R.A. and Fuller, J.J. Navy technical manual system (NTMS) program summary 
(TM-I86-76-I). Bethesda, MD: Naval Ship Research and Development Center, March 1976. 

Swaney, J.H., Janik, C, Bond, S., and Hayes, J.R. Editing for comprehension: 
Improving the process through reading protocols. (DDP TR-14). Pittsburgh, Pa.: 
Communications Design Center, Carnegie-Mellon University. June 1981. 

Wright, P. Editing Policies and Processes. In Duffy, T.M. and Waller. R., eds. 
Designing Usable Texts. New York: Academic Press, I985. 



32 



30 

1 Portions of this work were funded by the Navy Personnel Research and Development 
Center, San Diego, California 

2 Requests for reprints should be sent to: T. Duffy, Communications Design Center, 
Carnegie-Mellon University, Pittsburgh. Pa. 15213 



33 



31 



Figure Captions 



Figure 1 - Typical Technical Information Development Process 
(from Deardorff et al., 1979) 
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